Summary

Closes #38. The Rust chunker resolved grammars through a curated ~14-grammar static set (language_for), so files in recognized-but-uncurated languages (kotlin, swift, php, scala, lua, …) were walked and indexed but fell through to line chunking instead of AST chunking — a real behavioral narrowing vs upstream semble, which uses tree_sitter_language_pack (≈all languages).

This swaps the curated set for the Rust tree-sitter-language-pack crate (306 grammars, full upstream parity — the Rust sibling of the package semble itself uses). 264 of csp's 265 EXTENSION_TO_LANGUAGE names now AST-chunk; only wolfram lacks a pack grammar.

Changes

• language_for → tree_sitter_language_pack::get_language(name).ok() — downloads the parser from GitHub releases on first use, caches it on disk; unknown language or offline fetch failure → None → line fallback (the prior degradation contract, unchanged).
• is_supported_language → has_language(name) — a metadata-only lookup (no download), so chunk_source gates AST chunking cheaply before paying for a fetch.
• Replaced 14 individual tree-sitter-* crates with tree-sitter-language-pack = "1.9". tree-sitter stays a direct dep and resolves to the same 0.26.x (ABI-compatible Language).
• ADR-0004 records the single-binary ↔ runtime-grammar-cache trade-off; semble.md §4.3/§4.5/§6.2 updated.

Trade-off (decided with maintainer, ADR-0004)

The csp binary stays a single executable, but AST chunking is no longer fully offline/self-contained — grammars fetch from GitHub releases on first use and cache under the OS cache dir. Degradation is graceful (offline → line chunking, exactly what an unsupported language already did) and no language regresses below prior behavior. Binary shrinks (no compiled-in grammars).

Test plan

• cargo fmt --all ✅ · cargo clippy --all-targets --all-features -- -D warnings ✅ · cargo test --workspace ✅ (256 + 8 passed, 4 ignored)
• Offline-green: recognition (has_language) + fallback assertions run in the default suite.
• Real-parse tests are #[ignore]d (need network to download a grammar). Verified locally:

  cargo test -p csp chunking -- --ignored   # 3 passed (rust, multibyte, kotlin/swift/php AST-chunk)
  

Acceptance criteria (#38)

• Target grammar set decided & documented (ADR-0004, with binary-size/offline rationale).
• Selected languages return Some(Language) from language_for and AST-chunk (verified by newly_supported_languages_are_ast_chunked).
• Quality gate green.
• semble.md §6.2 item updated/closed.

Note: independent of #37 (chunk-length 750 / ranking) — branches off main, touches disjoint lines.

Summary by cubic

Switches the Rust chunker to tree-sitter-language-pack for near-full grammar coverage; 264/265 recognized languages now AST-chunk instead of lines. Parsers download on first use and cache on disk, with one-time warnings on grammar load failures and graceful line fallback when offline.

• New Features

  • Resolve grammars via tree_sitter_language_pack::get_language(name); is_supported_language uses has_language(name) (metadata-only, no download).
  • Coverage: 306 grammars; only wolfram in EXTENSION_TO_LANGUAGE still line-chunks.
  • Failure behavior: unknown/offline → None + one-time stderr warning per language → line fallback.
  • CI: add a network-gated ignored-tests job (manual/weekly) to run real parse tests that download grammars.

• Dependencies

  • Replaced 14 tree-sitter-* crates with tree-sitter-language-pack = "1.9"; tree-sitter remains (0.26.x ABI-compatible).
  • Trade-off documented in ADR-0004 (single binary with runtime grammar cache).

^{Written for commit 60530fd. Summary will update on new commits.}

Summary by CodeRabbit

• Documentation

  • Added an ADR and updated the documentation to explain the new grammar-resolution and chunking behavior (including offline fallback).

• Refactor

  • Expanded AST chunking coverage to hundreds of languages via a centralized language-pack source.
  • Reduced package/binary size by removing bundled grammars.
  • Grammars are downloaded on first use and cached; when unavailable/offline, chunking falls back to line-based behavior.

• Tests

  • Adjusted CI to run ignored/network-dependent parsing tests only on manual or scheduled runs.